Your flaky tests might be time dependent

Have you ever dealt with unexpected test failures, and no matter how many times you re-run the tests, they still fail? You assume that you introduced a bug, so you stash your changes and checkout main and run the same tests, but are surprised to see that they still fail, even though the latest build is passing. Dejected, you close your laptop and call it a day, only to return to a passing test suite the next morning. The problem wasn’t the codebase, but the test suite. More specifically, the problem was with when the test suite was run.

A Simple Example

Image you’re working on a new feature that prevents notifications from being created outside business hours. Here’s what the current state of the codebase looks like.

# app/controllers/notifications_controller.rb
def create
  @notification = Notification.new(notification_params)

    if @notification.save
      ...
    else
      ...
    end
  end
end

# test/controllers/notifications_controller_test.rb
test "should create notification" do
  assert_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end

You already have existing test coverage for creating a notification, so you go ahead and write some tests and build the new feature.

# app/controllers/notifications_controller.rb
def create
  @notification = Notification.new(notification_params)

    if within_business_hours? && @notification.save
      ...
    else
      ...
    end
  end

  private

  def within_business_hours?
    Time.use_zone("America/New_York") do
      Time.current.between?(
        Time.zone.parse("06:00"),
        Time.zone.parse("17:59")
      )
    end
  end
end

# test/controllers/notifications_controller_test.rb
test "should not create notification before business hours" do
  travel_to(Time.use_zone("America/New_York") { Time.parse("2022-11-19T05:59") })

  assert_no_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end

test "should not create notification after business hours" do
  travel_to(Time.use_zone("America/New_York") { Time.parse("2022-11-19T18:00") })

  assert_no_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end

test "should create notification" do
  assert_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end

All three tests pass, so you create a pull request and request a review. Later that day, your co-worker in California pulls down your branch, but is not able to get the test suite to pass locally.

The problem is that it’s 3:00 PM local time for your co-worker, which is 6:00 PM on the East Coast, meaning it’s after business hours. You never ran into this problem during feature development because you are on the East Coast, and work within the app’s business hours.

Fortunately, your West Coast co-worker realizes the problem and updates the existing test.

# test/controllers/notifications_controller_test.rb
test "should create notification within business hours" do
  travel_to(Time.use_zone("America/New_York") { Time.parse("2022-11-19T06:00") })

  assert_difference("Notification.count") do
    post notifications_url, params: {...}
  end
end

How do I know if my test suite is time dependent?

The previous example isn’t dissimilar to when a test suite begins to fail after a change in Daylight Savings. The root cause is that the tests are time dependent, but the symptoms don’t present themselves immediately. It’s not until you are working during an unusual time, or a new team member joins from another time zone, that the symptoms appear. This is even more relevant when you’re on a globally distributed team.

One method to check if your test suite is time dependent is to temporarily change your system clock to a time outside normal business hours, and then run the test suite looking for any new failures. A better approach would be to leverage Rails’ existing travel_to method and set an environment variable before running the test suite.

# test/test_helper.rb
class ActiveSupport::TestCase
  ...
  if ENV["TRAVEL_TO"]
    def setup
      travel_to Time.use_zone(ENV.fetch("TIME_ZONE", "America/New_York")) {
        Time.parse(ENV["TRAVEL_TO"])
      }
    end
  end
end

TRAVEL_TO=2022-11-18T05:00 rails test

The above example would run all tests at 5AM local time in the East Coast. This approach can be improved by creating a binstub to run the test suite against each hour of the day.

#!/usr/bin/env ruby
require "time"

failures = []

0.upto(24) do |i|
  hour = sprintf("%02d", i)
  date = "2022-11-19T#{hour}:00"

  puts "\n== Testing with #{date} =="

  unless system("TRAVEL_TO=#{date} rails test")
    failures << Time.parse(date).strftime("%I:%M%p %Z")
  end
end

if failures.any?
  puts "\n== The following times of day resulted in failures =="
  puts failures
end

Had we run this binstub in our example from earlier, we would have seen that our test suite was time dependent between 6:00AM and 5:59PM local time EST.

== The following times of day resulted in failures ==
12:00AM EST
01:00AM EST
02:00AM EST
03:00AM EST
04:00AM EST
05:00AM EST
06:00PM EST
07:00PM EST
08:00PM EST
09:00PM EST
10:00PM EST
11:00PM EST
12:00AM EST

I would encourage you to occasionally run this script against your test suite in an effort to catch unexpected time dependencies before they prevent an early morning deploy due to CI failures.